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ABSTRACT 

Homework has been a mysterious world to educators due to the 
fact that it is hard to collect data with regard to homework 
behaviors. Little is known about when a student works on 
homework, how long it takes him to complete the homework, how 
much time he spends on a problem and whether and where he has 
struggled, etc. Such information not only have implications on a 
student’s performance level on assigned skills, but also are 
potential indicator of his non-cognitive status, such as engagement 
with homework and whether he was persistent. In this paper, we 
present our initial effort to uncover the mysterious world through 
exploratory analyses of the system logs from the ASSISTments 
platform when 690 7th grade students in the state of Maine did 
their math homework in the system. 
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1. INTRODUCTION 

Homework is a well-established practice in schools, despite all the 
controversial discussion regarding its influence on learning 
(Kohn, 2006), and the research knowledge base for the 
effectiveness of homework is also well established (Cooper et al., 
2006). Yet, without explicit interventions, homework has been 
commonly underutilized for improving teaching and learning. 
Educational technologies have gained popularity in schools (e.g., 
Khan Academy, DreamBox, IXL.com), but not at home. Most of 
the computer programs for homework are for college-level 
populations (e.g., WebAssign, Mastering Physics, OWL), but not 
in K-12 settings. Homework has been a mysterious world to 
educators partly due to the fact that it is hard to collect data. 
However, information from homework, such as when a student 
works on homework, how long it takes him to complete the 
homework, how much time he spends on a problem and whether 
he has struggled, has not only implications on a student’s 
performance level on assigned skills, but also is potential indicator 
of his non-cognitive status, such as engagement with homework 
and whether he was being persistent. 

2. BACKGROUND 

ASSISTments ( www.assistments.org) is an online tutoring system 


that provides “formative assessments that assist.” Teachers choose 
(or add) homework items in ASSISTments and students can 
complete their homework items online. As students do homework 
in ASSISTments, they receive feedback on the correctness of their 
answers. Some problem types also provide hints on how to 
improve their answers, or help decompose multistep problems 
into parts. Teachers receive reports on their students’ homework 
and can use this information to organized more targeted 
homework reviews, to assign specific follow-up work to particular 
students, and to more generally adapt or differentiate their 
teaching. 

Prior research also has established the promise of ASSISTments 
for improving student outcomes in middle school mathematics 
through homework support (Mendicino et al., 2009; Singh et al., 
2011; Kelly et al., 2013). Building on this prior work, a large- 
scale efficacy study is being conducted with ASSISTments in the 
state of Maine where a one-to-one laptop program was well 
established, to evaluate the efficacy of ASSISTments for online 
homework support. This randomized controlled trial involves 45 
middle school schools that were randomly assigned to treatment 
or control (i.e. “business as usual”) conditions. The intervention is 
implemented in Grade 7 math classrooms in treatment schools 
over 2 consecutive years. In the treatment condition, teachers 
receive professional development and use ASSISTments to assign 
homework for their students during the school year. 

3. METHOD 
3.1 Data 

For this study, we collected homework log of 690 7 lh grade 
students from classes of 17 teachers in 9 middle schools that 
participate in the efficacy study. The data set includes 779 
homework assignments made by the teachers during January and 
February 2014. These students have been using ASSISTments to 
do their homework since the beginning of the school year and 
their teachers started using ASSISTments since September 2012. 
We excluded the problems that took students over 10 minutes to 
complete, considering students were likely to be off-task and thus 
the measure of completion time might not accurate. On average, 
each student solved 181 problems, and the number varies a lot 
among students (standard deviation = 163). In addition to student 
homework log, we also collected teacher’s usage data, in 
particular, when they have opened a report provided by 
ASSISTments. 

Based on the student log and teacher usage data, we calculated the 
following metrics 

• %Correct—student’s average percent correct on all 
problems in an assignment 
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• AvgAttempt—the average number of attempts 1 a 
student made on a problem in an assignment 

• AvgFirstResponseTime—the average amount of time it 
took a student to respond to a problem in an assignment 

• AvgTotalTime—the average total time it took a student 
to complete a problem in an assignment 

• StartHour—the hour of the day when the student started 
working on an assignment 

• Completionlndicator—whether an assignment was 
completed on time, late or not completed. 

• CompletionRate—a student’s overall homework 

completion rate during the time period 

• %ReportOpening—a teacher level metric, the 

percentage of assignments for which a teacher has 
opened related ASSISTments reports. For example, if a 
teacher has made 10 homework assignments to her 
students, but only looked at reports for 4 of the 
assignments, then %ReportOpening will be 40%. 

3.2 Analysis and Findings 

Our analysis was mostly exploratory. First, we plotted the data 
(see Figure 1) to see when students started working on homework, 
and if there is any association between when a student started and 
whether the assignment was completed on time or not. We 
observed that for the 8573 instances of assignments that were 
completed on time, most of the time students started around 1 lam, 
or 12pm, or early in the morning at 9am. The assignments that 
were not completed tended to start a bit later at 1pm or 10am. 



Figure 1. Time students start working on homework 

Then we looked to see whether there was any difference in 
student’s performance or behaviors when they completed 
homework assignments on time or not. We found that for 
assignments that were not completed, students were significantly 
(unpaired t-test, p < .01) low on %Correct metric, yet high on 
AvgFirstResponseTime, and AvgTotalTime, comparing to their 
performance on assignments that were completed on time, 
indicating students were struggling with the problems in those 
assignments. Meanwhile, students were also significantly low 
(upaired t-test, p < .01) on AvgAttempt, suggesting they were not 
as persistent when trying to solve the problems. 

Teacher’s review of homework performance report is a critical 
step in the ASSISTments logic model and teachers are encouraged 
to look at the reports to direct their homework review with 
students and adapt their instructions. During the interviews 
(another data collection activity of the efficacy study), teachers 
indicated homework review time has been largely reduced 
because of that the ASSISTments reports have made the review 
more targeted. While we don’t have the classroom observation 


1 In ASSISTments, students are allowed to make multiple attempts at 
problems until they solve the problem correctly. 


data yet, we consider %ReportOpening as an indicator of how 
often the homework review was done. We discretized 
%ReportOpening into 3 bins: low, medium and high, and 
aggregated other metrics across students within each bin. We 
found in the bin where %ReportOpening was low, students’ 
average %Correct and CompletionRate were significantly higher 
yet AvgFirstResponseTime and AvgTotalTime were all 
significantly lower, comparing to those for the “high” bin. While 
this finding was against our initial instinct, it is too early to draw 
any conclusion regarding a casual relationship between teacher’s 
review practices and student’s homework performance from this, 
given that the analysis wasn’t tracking changes in the same 
teacher’s classes longitudinally, and didn’t account for any 
incoming homework performance data of the students (e.g. 
homework completion rate, %Correct, etc). Teachers who knew 
their students had problems with completing homework may 
choose to look at reports more often to monitor student’s progress. 

4. CONCLUSION 

In this paper, we presented some initial results from analyzing 
student homework logs and teacher’s usage of an online 
homework support program as a part of an efficacy study. The 
analyses here represent the beginning of our efforts to understand 
the world of homework. In the future, we plan to link student’s 
homework log data with their unit test scores (as proximal 
measure of their knowledge) and end of year standardized test 
scores to investigate the relationship between homework and 
learning outcomes. We also plan to analyze student homework log 
data, teacher’s report usage data and test scores together 
longitudinally and triangulate the results with findings from field 
classroom observations to further investigate the impact of 
teacher’s review practices on student’s learning outcome and on 
how students do their homework. 
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